Pipeline as Code

Fabrice Flore-Thébault

2017-03-30

About me

inspiration/quebec-kayaking-scene_crop.jpg

Topics

inspiration/jenkins/2016-survey-mission-critical.png

2016 Jenkins community survey results

inspiration/jenkins.io/files/2016-jenkins-community-survey.png

Pipelines color as code … learn from the metaphor

inspiration/flickr/telstar-4883861519-1024.jpg

Pipelines elements as code … learn from the metaphor

inspiration/engineerpkorg/piping-codes.jpg

Pipeline flow concept

inspiration/jenkins/realworld-pipeline-flow.png
Figure 5. Image source: Jenkins Handbook https://jenkins.io/doc/book/pipeline/

* as Code: Automation is for people

inspiration/ansible/Future-of-Automation-2.jpg

History of (Declarative) Pipeline as code in Jenkins

inspiration/jenkins.io/images/post-images/blueocean-dev-log/creating-pipeline-from-github.png
Figure 7. Image source: http://jenkins.io

Declarative Pipeline Editor in Blue Ocean

inspiration/jenkins.io/images/post-images/blueocean/pipeline-editor.png
Figure 8. Image source: http://jenkins.io

Declarative Pipeline vs. Scripted Pipeline

Declarative Pipeline Scripted Pipeline

On top of Scripted Pipeline

Full power of Groovy scripting language

Consistent structure

Greater liberty

Human readable

Complex

Lower barrier entry

Groovy "Expert" needed

UI Editor including syntax validation

Frequent obscure syntax errors

Validation of syntax at very beginning of a build

Error during the build

Error point to the point in the pipeline which is causing problem

Difficult to find source of error

inspiration/jenkins.io/images/post-images/blueocean/pipeline-run.png
Figure 9. Image source: http://jenkins.io

One step back … Pipelines in the CI/CD tools

inspiration/jamesbowman/ContinuousDeliveryToolLandscape.jpeg

AWS CodePipeline - concept

inspiration/aws/PipelineFlow.png

AWS CodePipeline - tools

inspiration/aws/codepipeline-diagram.png
Figure 12. Image source: https://aws.amazon.com/codepipeline/

Atlassian: Bitbucket (SAAS) has Pipelines, but not Bamboo

inspiration/bitbucket/screen2.png

Platform.sh: build and deploy processes hooks as code

inspiration/platform.sh/build-pipeline.png

Getting started on Jenkins: Install suggested plugins

inspiration/jenkins.io/images/getting-started-setup.png
Figure 15. Image source: http://jenkins.io

Jenkins new UI: Blue Ocean

inspiration/jenkins.io/images/blueocean/blueocean-successful-pipeline.png
Figure 16. Source: http://jenkins.io

Interactions in Blue Ocean

inspiration/jenkins.io/images/blueocean/input-step.png
Figure 17. Image source: http://jenkins.io

Goodbye old interface ?

inspiration/jenkins.io/images/pipeline-stage-view.png
Figure 18. Image source: http://jenkins.io

Jenkins Declarative Pipeline

One example: Declarative Pipeline to build this presentation
pipeline {
  agent any
  stages {
    stage('Build') {
      steps {
        sh '''
          rm -f *.html
          asciidoc --backend list | grep deckjs ||  asciidoc  --backend install deckjs-1.6.3.zip
          for file in *.asciidoc
          do
            asciidoc $file
          done
        '''
      }
    }
    stage('Tests') {
      steps {
        sh 'file pipeline-as-code.html'
      }
    }
    stage('Deploy') {
      steps {
        echo 'Hello World'
      }
    }
  }
  post {
    always {
      archive '*.html'
      echo 'I will always say Hello again!'
    }
  }
}

Improvements

Source: https://jenkins.io/blog/2017/03/16/fosdem-event-report/

pipeline: this makes it a declarative pipeline

pipeline {
  agent {
    docker "fstab/asciidoc"
  }
}

environment, options and parameters

pipeline {
  agent any
  environment {
    DISABLE_AUTH = 'true'
    DB_ENGINE    = 'sqlite'
    AWS_ACCESS_KEY_ID     = credentials('AWS_ACCESS_KEY_ID')
    AWS_SECRET_ACCESS_KEY = credentials('AWS_SECRET_ACCESS_KEY')
  }
  options {
    buildDiscarder(logrotator(numToKeepStr:'5'))
    timeout(time: 30, unit: 'MINUTES')
  }
  parameters {
    string(name: 'TARGET',
           description: "Where we're deploying to",
           )
  }
  stages {
    stage('Build') {
      steps {
        sh 'printenv'
      }
    }
  }
}

stages: Running multiple steps

pipeline {
    agent any
    stages {
        stage('Build') {
            steps {
                sh 'echo "Hello World"'
                sh '''
                    echo "Multiline shell steps works too"
                    ls -lah
                '''
            }
        }
    }
}

retry, timeout, condition

pipeline {
    agent any
    stages {
        stage('Deploy') {
            steps {
                when {
                  branch "*/master"
                }

                retry(3) {
                    sh './flakey-deploy.sh'
                }

                timeout(time: 3, unit: 'MINUTES') {
                    sh './health-check.sh'
                }
            }
        }
    }
}

post: finishing up

pipeline {
    agent any
    stages {
        stage('Test') {
            steps {
                sh 'echo "Fail!"; exit 1'
            }
        }
    }
    post {
        always {
          echo 'This will always run'
          archive '*.html'
        }
        success {
            echo 'This will run only if successful'
        }
        failure {
            echo 'This will run only if failed'
            mail to: 'team@example.com',
                 subject: "Failed Pipeline: ${currentBuild.fullDisplayName}",
                 body: "Something is wrong with ${env.BUILD_URL}"
        }
        unstable {
            echo 'This will run only if the run was marked as unstable'
        }
        changed {
            echo 'This will run only if the state of the Pipeline has changed'
            echo 'For example, if the Pipeline was previously failing but is now successful'
        }
    }
}

Scripted Pipeline build blocks: heavyweight vs. flightweight

// Wait outside an executor context
input "Do you want to continue?"

node('php') {
  // Do something heavy
}

https://jenkins.io/doc/book/pipeline/syntax/

Scripted Pipeline build blocks: try … catch … finally

try {
  // do something
}
catch (e) {
  // In case of failure
  throw e
}
finally {
  // Success or failure, always do ...
}

Scripted Pipeline build blocks: abort obsolete jobs

milestone 10
notify_build('STARTED')
build_project()
milestone 20
deploy_now('dev')
milestone 30

Scripted Pipeline: define the workflow

/*
 * The workflow: Gitflow.
 * On branch develop, deploy automatically to development.
 * On master branch, deploy automatically to QA, and wait for a validation before deploy to production.
 * Obsolete builds are canceled.
 * Externalize the content of the build and deploy tasks.
 */

try {
  node('php') {
    milestone 10
    notify_build('STARTED')
    build_project()
    milestone 20
    switch ( "${env.BRANCH_NAME}" ) {
      case "develop":
        milestone 30
        deploy_now('dev')
        milestone 40
        return
      case "master":
        milestone 30
        deploy_now('stg')
        milestone 40
        return
    }
  }
  switch ( "${env.BRANCH_NAME}" ) {
    case "master":
      milestone 60
      deploy_after_validation('prd')
      milestone 80
      return
  }
}
catch (e) {
  currentBuild.result = "FAILED"
  throw e
}
finally {
  // Success or failure, always send notifications
  notify_build(currentBuild.result)
}

Scripted Pipeline: The build step

/*
 * Define how to build a project.
 * We assume we are already in a node context.
 */
def build_project() {
  stage ("Build") {
    // get some information about the build environment
    echo "branch name: ${env.BRANCH_NAME}"
    sh "env"
    // Get versioned files
    checkout scm
    // Build -- adapt to your project
    dir('www') {
      sh "sh ../.jenkins/build.sh"
      archiveArtifacts(
          artifacts: '**',
          fingerprint: true,
          onlyIfSuccessful: true
      )
    }
  }
}

Scripted Pipeline: the deploy step

/*
 * Define how to deploy a project without waiting.
 * We assume we are already in a node context.
 * Deployment on corresponding environment will be attempted  only if the file exist in the '.jenkins/config' directory:
 * - dev.groovy
 * - tst.groovy
 * - stg.groovy
 * - prd.groovy
 */

def deploy_now(String targetEnv = 'dev') {
  stage ("Deploy to ${targetEnv}") {
    go = 'undefined'
    try {
      // load parameters for the environment ${targetEnv}
      load path: ".jenkins/${targetEnv}.groovy"
      echo "Variables defined for deployment in .jenkins/${targetEnv}.groovy : " +
          "credentialsID=${credentialsId}, " +
          "server=${target_server}, " +
          "user=${target_user}"
      go = 'true'
    }
    catch (e) {
      echo "No deployment done since the environment is not defined in .jenkins/${targetEnv}.groovy"
      go = 'false'
    }
    //echo "${go}"
    if ( "${go}"== 'true' ) {
      ansiblePlaybook(
          credentialsId: "${credentialsId}",
          extras: "--user=${target_user} --verbose",
          installation: "ansible",
          inventory: "${target_server},",
          playbook: ".jenkins/deploy.yml",
          sudoUser: null
      )
    }
  }
}

Scripted Pipeline as code: Jenkinsfile - the validation step

/*
 * Define how to wait for user validation before launching a deploy.
 * We assume we are outside of a node context.
 */
def deploy_after_validation(String targetEnv = 'dev') {
  stage ("Validate to deploy to ${targetEnv}") {
    timeout(time: 100, unit: 'DAYS') {
      hipchatSend(
          color: 'PURPLE',
          message: "User input requested : Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]' (${env.BUILD_URL})"
      )
      input "Deploy to ${targetEnv}?"
    }
  }
  node('php') {
    deploy_now("${targetEnv}")
  }
}

Some usefull lectures

Challenges

inspiration/jedi.be/devops-areas-cams.png

/